Interactive Language-Based Task Library Instruction and Management for Single and Multiple Robots

Robots acquire behaviors to perform tasks, in general by being programmed, or occasionally by being instructed through demonstrations. In this thesis, we address the challenge of providing task behaviors to a robot through language instructions and interactions. We consider robots equipped with built-in motion and perception primitives with their functionality known to the user. We contribute Instruction Graphs, a robot task representation for composing sequences, conditionals, and loops of robot primitives. Instruction Graphs are only robot-primitive dependent, therefore applying to any robot hardware platform. We present the process by which Instruction Graphs are incrementally created from language instructions, as well as the method to execute Instruction Graphs on robots. We present examples and demonstrations of acquired Instruction Graphs with our CoBot mobile service robot and our Baxter manipulator robot. We extend Instruction Graphs from single-robot tasks to multi-robot tasks with multi-robot sparse interaction coordination. We introduce Instruction Graph constructs that enable a robot to query the state of another robot, similar to a sensing primitive. We show examples where multi-robot Instruction Graphs are used to coordinate the two arms of Baxter, and to coordinate CoBot and Baxter. Motivated by the fact that the robots can accumulate a large number of Instruction Graphs, we finally address the problem of managing a task library. We first devise an approach for correcting single steps of an Instruction Graph from a task library during its test execution. We then contribute an algorithm for learning Generalized Instruction Graphs, which represent parameterized tasks, from a task library of instantiated Instruction Graphs. We further contribute an algorithm that, given the initial steps of a new task, proposes an autocompletion based on a recognized similar Generalized Instruction Graph. We show results with large task libraries, in particular for the Baxter robot, in which the instruction of new tasks benefits from this generalization and autocompletion approach. We discuss future work in terms of achieving the complete deployment of our approach on multiple robot platforms as available. We also discuss conditional representations of robot primitives, alternative filters for generalization and autocompletion, and the potential to interleave human instruction, multi-step correction, and the automated planning of Instruction Graphs.